Papers with encoder models
Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing work proposes dialect adaptation for encoder models or encoder-decoder models. |
| Approach: | They propose to use MD-3 to combine task adapters and dialect adapters to decoder models using a masked word game-playing conversation. |
| Outcome: | The proposed architecture outperforms baselines on Indian English and Nigerian English on a masked conversation with two models. |
Assessing In-context Learning and Fine-tuning for Topic Classification of German Web Data (2024.acl-srw)
Copied to clipboard
| Challenge: | Using a few hundred annotated data points per topic, we detect content related to three German policies in a database of scraped webpages. |
| Approach: | They propose to use annotated data to train a binary classification task to detect topic-related content in a scraped database of webpages. |
| Outcome: | The proposed model detects content related to three German policies in a scraped database of scrapes of webpages using a few hundred annotated data points per topic. |
Nuanced Toxicity Detection in Spanish: A New Corpus and Benchmark Study (2026.findings-eacl)
Copied to clipboard
Alba María Mármol-Romero, Robiert Sepúlveda-Torres, Estela Saquete, María-Teresa Martín-Valdivia, L. Alfonso Ureña
| Challenge: | Existing corpora for Spanish are under-resourced for toxic content detection . sarcasm, indirect aggression, irony, and other toxicity are not detected in English . |
| Approach: | They propose to extend the NECOS-TOX corpus to include 4,011 Spanish comments . each comment is annotated across three levels of toxicity, with substantial inter-annotator agreement . |
| Outcome: | The proposed model performs on par with larger models and is released publicly . the proposed model is based on a human-in-the-loop active learning strategy . |
Knowledge Augmentation Enhances Token Classification for Recipe Understanding (2026.eacl-long)
Copied to clipboard
| Challenge: | Using entity type-specific and knowledge-augmented token classification, we achieve state-of-the-art (SOTA) results on 5 out of 7 benchmark recipe datasets, significantly outperforming traditional token classification methods. |
| Approach: | They propose an entity type-specific and knowledge-augmented token classification framework to improve encoder models’ performance on recipe texts. |
| Outcome: | The proposed model outperforms traditional token classification methods on 5 out of 7 recipe datasets and is the largest annotated food-related dataset to date. |
Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve robustness require changing the fine-tuning process or large-scale data augmentation, which are infeasible or cost prohibitive for closed-source models. |
| Approach: | They propose to prioritize more complex examples or replace existing training examples with LLM-generated data to improve performance on OOD NLI datasets. |
| Outcome: | The proposed methods improve performance on difficult OOD datasets while training with synthetic data leads to substantial improvements on easier OOD data. |
FactAppeal: Identifying Epistemic Factual Appeals in News Media (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods focus on the content of factual statements and ignore the epistemic structures that confer credibility and persuasive force to these claims. |
| Approach: | They propose a task of Epistemic Appeal Identification to identify whether and how factual statements have been anchored by external sources or evidence. |
| Outcome: | The proposed task identifies whether and how factual statements have been anchored by external sources or evidence. |
Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metric (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing toxicity metrics rely on encoder models trained on specific toxicity datasets, which are susceptible to out-of-distribution (OOD) problems and depend on the dataset’s definition of toxicity. |
| Approach: | They propose a robust metric grounded on LLMs to flexibly measure toxicity according to the given definition by analysing toxicity factors and intrinsic toxic attributes. |
| Outcome: | The proposed metric improves on conventional metrics by 12 points in the F1 score and shows that upstream toxicity significantly influences downstream metrics, suggesting that LLMs are unsuitable for toxicity evaluations within unverified factors. |
Learning Semantic Structure through First-Order-Logic Translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that transformer-based language models can confuse which predicates apply to which objects . a this is a crucial building block of semantic structure, but if an LM mixes up which objects have which property, it makes errors in reasoning . |
| Approach: | They propose to use transformer-based language models to learn predicate argument structure from simple sentences. |
| Outcome: | The proposed model can learn predicate argument structure from simple sentences. |
Towards Long Context Hallucination Detection (2025.findings-naacl)
Copied to clipboard
Siyi Liu, Kishaloy Halder, Zheng Qi, Wei Xiao, Nikolaos Pappas, Phu Mon Htut, Neha Anna John, Yassine Benajiba, Dan Roth
| Challenge: | Large language models are prone to contextual hallucination, generating information that is either unsubstantiated or contradictory to the given context. |
| Approach: | They propose a dataset specifically designed for long-context hallucination detection. |
| Outcome: | The proposed architecture outperforms existing models while providing faster inference. |
SuperGLEBer: German Language Understanding Evaluation Benchmark (2024.naacl-long)
Copied to clipboard
| Challenge: | a new set of German-pretrained models are being released, but no established, diverse and systematic evaluation suite is available for them. |
| Approach: | They assemble a Natural Language Understanding benchmark suite for the German language and evaluate 10 existing German-pretrained models. |
| Outcome: | The proposed benchmark suite evaluates 10 German-pretrained models on 29 tasks . the results show that encoder models are good choices for most tasks, but not all . |
ltzGLUE: Luxembourgish General Language Understanding Evaluation (2026.findings-acl)
Copied to clipboard
Alistair Plum, Felicia Körner, Anne-Marie Lutgen, Laura Bernardy, Fred Philippy, Emilia Milano, Nils Rehlinger, Cedric Lothritz, Tharindu Ranasinghe, Barbara Plank, Christoph Purschke
| Challenge: | ltzGLUE is the first official NLU benchmark for Luxembourgish (LTZ) based on the popular GLUE benchmark for English. |
| Approach: | They propose a new natural language understanding (NLU) benchmark for Luxembourgish based on the popular GLUE benchmark for English. |
| Outcome: | The proposed model performs well across many languages and is based on the GLUE benchmark for English. |
A Lightweight Method to Generate Unanswerable Questions in English (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to build robust question answering models are too complex . antonym and entity swaps on answerable questions are used to build models . |
| Approach: | They propose a method for performing antonym and entity swaps on unanswerable questions. |
| Outcome: | The proposed method outperforms the previous state-of-the-art and has higher human-judged relatedness and readability. |
PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing retrieval-based methods to pre-select tools are often misaligned with tool-calling LLMs due to separate training processes. |
| Approach: | They propose a method to fine-tune retrievers to find useful tools by using a frozen LLM. |
| Outcome: | The proposed method fine-tunes retrievers to find useful tools using a frozen LLM . it improves tool selection accuracy and can be generalized to new queries and tools . |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |
Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models (2025.findings-acl)
Copied to clipboard
Ahmed Elshabrawy, Thanh-Nhi Nguyen, Yeeun Kang, Lihan Feng, Annant Jain, Faadil Abdullah Shaikh, Jonibek Mansurov, Mohamed Fazli Mohamed Imam, Jesus-German Ortiz-Barajas, Rendi Chevi, Alham Fikri Aji
| Challenge: | Large Language Models excel in zero-shot and few-shot tasks, but their architecture makes them difficult to use. |
| Approach: | They adapt Large Language Models (LLMs) for zero-shot generalization using Statement Tuning . they find encoders can achieve zero- shot cross-lingual generalization . |
| Outcome: | The proposed model generalizes well across languages while being more efficient. |
Probing Representations for Document-level Event Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level information extraction tasks require a more comprehensive understanding that often extends to the entire input document. |
| Approach: | They propose to use probing to analyze document-level information extraction representations by embedding probes into a standard dataset. |
| Outcome: | The proposed models improve argument detections but struggle with document length and cross-sentence discourse. |
Language Models Lack Temporal Generalization and Bigger is Not Better (2025.findings-acl)
Copied to clipboard
| Challenge: | 450 encoder models are fine-tuned on 15 data splits on a task to detect events in Early Modern Dutch archival texts. |
| Approach: | They propose to fine tune six encoder models that have been pretrained with very different data on a task in Early Modern Dutch archival texts. |
| Outcome: | The proposed model is fine tuned with 5 seeds on 15 different data splits and reaches highest F1 performance. |
Overcoming Black-box Attack Inefficiency with Hybrid and Dynamic Select Algorithms (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing black-box attack methods require large number of queries, making them inefficient and impractical for researchers. |
| Approach: | proposed new attack selection strategies combine strengths of previous selection algorithms . Hybrid Select merges generalized BinarySelect techniques with GreedySelection . Dynamic Select learns which lengths of texts each selection method should be applied to . |
| Outcome: | proposed methods combine the strengths of previous selection algorithms to improve performance. |
Cut the Deadwood Out: Backdoor Purification via Guided Module Substitution (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Model NLP models are often trained on datasets from untrusted platforms, posing significant risks of data poisoning attacks. |
| Approach: | They propose a retraining-free method that selectively replaces modules in the victim model based on a trade-off signal between utility and backdoor. |
| Outcome: | The proposed method outperforms even the strongest defense baseline against challenging attacks like LWS. |
SwissGov-RSD: A Human-annotated, Cross-lingual Benchmark for Token-level Recognition of Semantic Differences Between Related Documents (2026.acl-long)
Copied to clipboard
| Challenge: | Recognizing semantic differences across documents is crucial for text generation evaluation and content alignment . but as a standalone task, it has received little attention, a new study shows . evaluating semantic differences between documents is an underexplored challenge in natural language understanding . |
| Approach: | They introduce SwissGov-RSD, the first naturalistic, document-level, cross-lingual dataset for semantic difference recognition. |
| Outcome: | The proposed dataset shows that current approaches perform poorly on monolingual, sentence-level and synthetic benchmarks. |
Explanation Quality Assessment as Ranking with Listwise Rewards (2026.findings-acl)
Copied to clipboard
| Challenge: | a new approach to explanation quality assessment is to rank explanations by relative quality . standard reward objectives do not preserve graded distinctions well enough for policy optimization . |
| Approach: | They reformulate explanation quality assessment as a ranking problem instead of a generation problem . they train listwise and pairwise ranking models to preserve ordinal structure . |
| Outcome: | The proposed model outperforms regression on score separation and performance on listwise and pairwise models. |